Skip to content

Add wget wait time for retries #325

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 5 commits into
base: dev
Choose a base branch
from
Open

Add wget wait time for retries #325

wants to merge 5 commits into from

Conversation

@maxibor
Copy link
Member

@maxibor maxibor commented Nov 29, 2024

When downloading from EBI via ftp, I get frequent retries (both via wget, and via process retries) because of a failed: Connection refused.

Adding a waiting time seems to help a little bit

excerpt from wget manual

-w seconds
       --wait=seconds
           Wait the specified number of seconds between the retrievals.  Use of this option is recommended, as it lightens the
           server load by making the requests less frequent.  Instead of in seconds, the time can be specified in minutes using
           the "m" suffix, in hours using "h" suffix, or in days using "d" suffix.

           Specifying a large value for this option is useful if the network or the destination host is down, so that Wget can
           wait long enough to reasonably expect the network error to be fixed before the retry.  The waiting interval
           specified by this function is influenced by "--random-wait", which see.

PR checklist

  • This comment contains a description of changes (with reason).
  • Make sure your code lints (nf-core lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@maxibor maxibor changed the title Wget wait Add wget wait time for retries Nov 29, 2024
@github-actions
Copy link

github-actions bot commented Nov 29, 2024 •
edited
Loading

nf-core pipelines lint overall result: Passed ✅

Posted for pipeline commit ecffded

+| ✅ 165 tests passed       |+
#| ❔  15 tests were ignored |#

❔ Tests ignored:

  • files_exist - File is ignored: .github/workflows/awsfulltest.yml
  • files_exist - File is ignored: .github/workflows/awstest.yml
  • files_exist - File is ignored: assets/multiqc_config.yml
  • files_exist - File is ignored: conf/igenomes.config
  • files_exist - File is ignored: conf/igenomes_ignored.config
  • files_exist - File is ignored: conf/modules.config
  • files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
  • files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
  • files_unchanged - File ignored due to lint config: assets/nf-core-fetchngs_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-fetchngs_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-fetchngs_logo_dark.png
  • actions_ci - actions_ci
  • actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/fetchngs/fetchngs/.github/workflows/awstest.yml
  • multiqc_config - multiqc_config
  • modules_config - modules_config

✅ Tests passed:

Run details

  • nf-core/tools version 3.0.2
  • Run at 2025-03-07 12:07:12

@maxibor maxibor requested a review from drpatelh December 5, 2024 12:00
@nf-core-bot
Copy link
Member

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.0.2.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

@maxulysse maxulysse added this to the 1.13.0 milestone May 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@maxulysse maxulysse maxulysse approved these changes

@drpatelh drpatelh Awaiting requested review from drpatelh

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

1.13.0

Development

Successfully merging this pull request may close these issues.

4 participants

@maxibor @nf-core-bot @maxulysse @drpatelh